Using Scientific Abstracts to Measure Learning Outcomes in the Biological Sciences †

INTRODUCTION Educators must often measure the effectiveness of their instruction. We designed, developed, and preliminarily evaluated a multiple-choice assessment tool that requires students to apply what they have learned to evaluate scientific abstracts. This examination methodology offers the flexibility to both challenge students in specific subject areas and develop the critical thinking skills upper-level classes and research require. Although students do not create an end product (performance), they must demonstrate proficiency in a specific skill that scientists use on a regular basis: critically evaluating scientific literature via abstract analysis, a direct measure of scientific literacy. Scientific abstracts from peer-reviewed research articles lend themselves to in-class testing, since they are typically 250 words or less in length, and their analysis requires skills beyond rote memorization. To address the effectiveness of particular courses, in five different upper-level courses (Ecology, Genetics, Virology, Pathology, and Microbiology) we performed pre-and post-course assessments to determine whether students were developing subject area competence and if abstract-based testing was a viable instructional strategy. Assessment should cover all levels in Bloom's hierarchy, which can be accomplished via multiple-choice questions (2). We hypothesized that by comparing the mean scores of pre-and posttest exams designed to address specific tiers of Bloom's taxonomy, we could evaluate the effectiveness of a course in preparing students to demonstrate subject area competence. We also sought to develop general guidelines for preparing such tests and methods to identify test-and course-specific problems. PROCEDURE Two abstracts were selected by each instructor based on direct knowledge of the literature or after consulting content located in PubMed, Google Scholar, or other databases. Each exam, either A or B, consisted of the abstract and five multiple-choice questions relating to the abstract. See Appendix 1 for references used, sample abstracts, and questions. Within the first week of a class, students were given either exam A or B. Initial tests were collected, not discussed with the class, and graded by a centralized grader (see Appendix 1 for protocol). Approximately 80% into each course, students were given the alternate test such that all students took both tests. The means of pre-and posttest scores were computed. A dependent t-test analysis (paired samples) was performed to determine if the differences in pre-and posttest mean scores were significant; t-test analysis was also used to confirm that the A and B test versions did not produce statistically different means (i.e., to insure that neither the A nor the …